Stream: Internet Engineering Task Force (IETF) 


RFC: 9322 

Category: Standards Track 

Published: November 2022 

ISSN: 2070-1721 

Authors: T. Mizrahi F.Brockners S. Bhandari B. Gafni M. Spiegel 
Huawei Cisco Thoughtspot Nvidia Barefoot Networks 


RFC 9322 
In Situ Operations, Administration, and Maintenance 
(IOAM) Loopback and Active Flags 


Abstract 


In situ Operations, Administration, and Maintenance (IOAM) collects operational and telemetry 
information in packets while they traverse a path between two points in the network. This 
document defines two new flags in the IOAM Trace Option headers, specifically the Loopback 
and Active flags. 


Status of This Memo 


This is an Internet Standards Track document. 


This document is a product of the Internet Engineering Task Force (IETF). It represents the 
consensus of the IETF community. It has received public review and has been approved for 
publication by the Internet Engineering Steering Group (IESG). Further information on Internet 
Standards is available in Section 2 of RFC 7841. 


Information about the current status of this document, any errata, and how to provide feedback 
on it may be obtained at https://www.rfc-editor.org/info/rfc9322. 


Copyright Notice 


Copyright (c) 2022 IETF Trust and the persons identified as the document authors. All rights 
reserved. 


This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF 
Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this 
document. Please review these documents carefully, as they describe your rights and restrictions 
with respect to this document. Code Components extracted from this document must include 
Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are 
provided without warranty as described in the Revised BSD License. 
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1. Introduction 


November 2022 


IOAM [RFC9197] is used for monitoring traffic in the network by incorporating IOAM data fields 


into in-flight data packets. 
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IOAM data may be represented in one of four possible IOAM options: Pre-allocated Trace, 
Incremental Trace, Proof of Transit (POT), and Edge-to-Edge. This document defines two new 
flags in the Pre-allocated and Incremental Trace options: the Loopback and Active flags. 


The Loopback flag is used to request that each transit device along the path loops back a 
truncated copy of the data packet to the sender. The Active flag indicates that a packet is used for 
active measurement. The term "active measurement" in the context of this document is as 
defined in [RFC7799]. 


2. Conventions 


2.1. Requirements Language 


The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD 
NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to 
be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in 
all capitals, as shown here. 


2.2. Terminology 


Abbreviations used in this document: 


IOAM: In situ Operations, Administration, and Maintenance 


OAM: Operations, Administration, and Maintenance [RFC6291] 


3. New IOAM Trace Option Flags 


This document defines two new flags in the Pre-allocated and Incremental Trace options: 


Bit 1 "Loopback" (L-bit): When set, the Loopback flag triggers the sending of a copy of a packet 
back towards the source, as further described in Section 4. 


Bit 2 "Active" (A-bit): When set, the Active flag indicates that a packet is an active measurement 
packet rather than a data packet, where "active" is used in the sense defined in [RFC7799]. The 
packet may be an IOAM probe packet or a replicated data packet (the second and third use 
cases of Section 5). 


4. Loopback in IOAM 


The Loopback flag is used to request that each transit device along the path loops back a 
truncated copy of the data packet to the sender. Loopback allows an IOAM encapsulating node to 
trace the path to a given destination and to receive per-hop data about both the forward and 
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return paths. Loopback is intended to provide an accelerated alternative to Traceroute that 
allows the encapsulating node to receive responses from multiple transit nodes along the path in 
less than one round-trip time (RTT) and by sending a single packet. 


As illustrated in Figure 1, an IOAM encapsulating node can push an IOAM encapsulation that 
includes the Loopback flag onto some or all of the packets it forwards using one of the IOAM 
encapsulation types, e.g., [IOAM-NSH] or [IOAM-IPV6-OPTIONS]. The IOAM transit node and the 
decapsulating node both create copies of the packet and loop them back to the encapsulating 
node. The decapsulating node also terminates the IOAM encapsulation and then forwards the 
packet towards the destination. The two IOAM looped-back copies are terminated by the 
encapsulating node. 


+-------- + +-------- + +-------- + +-------- + +-------- + 
| | | LOAM | eee | SILOAM) | |(S3ee | IOAM | | | 
+-------- + +-------- + +-------- + +-------- + +-------- + 
| L2/L3 |<===>| L2/L3 |<===>| L2/L3 |<===>| L2/L3 |<===>| L2/L3 | 
+-------- + +-------- + +-------- + +-------- + +-------- + 

Source Encapsulating Transit Decapsulating Destination 

Node Node Node 
SS tliat IOAM-Domain ----------- > 


IOAM encap. with Loopback flag 
Data packet === >seSeSSSSSSSSSS5555555555S55=>----------- > 


| | 

IOAM looped back | 
i—i | 
IOAM looped back| 
€=SSSS2SSS52>S5S2S52>SSS=2=SS=S=+ 


Figure 1: Loopback in IOAM 


Loopback can be used only if a return path from transit nodes and destination nodes towards the 
source (encapsulating node) exists. Specifically, loopback is only applicable in encapsulations in 
which the identity of the encapsulating node is available in the encapsulation header. If an 
encapsulating node receives a looped-back packet that was not originated from the current 
encapsulating node, the packet is dropped. 


4.1. Loopback: Encapsulating Node Functionality 


The encapsulating node either generates synthetic packets with an IOAM trace option that has 
the Loopback flag set or sets the Loopback flag in a subset of the in-transit data packets. 
Loopback is used either proactively or on-demand, i.e., when a failure is detected. The 
encapsulating node also needs to ensure that sufficient space is available in the IOAM header for 
loopback operation, which includes transit nodes adding trace data on the original path and 
again on the return path. 
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An IOAM trace option that has the Loopback flag set MUST have the value '1' in the most 
significant bit of IOAM-Trace-Type and '0' in the rest of the bits of IOAM-Trace-Type. Thus, every 
transit node that processes this trace option only adds a single data field, which is the Hop_Lim 
and node_id data field. A transit node that receives a packet with an IOAM trace option that has 
the Loopback flag set and the IOAM-Trace-Type is not equal to '1' in the most significant bit and 
'0' in the rest of the bits MUST NOT loop back a copy of the packet. The reason for allowing only a 
single data field per hop is to minimize the impact of amplification attacks. 


IOAM encapsulating nodes MUST NOT push an IOAM encapsulation with the Loopback flag onto 
data packets that already include an IOAM encapsulation. This requirement is intended to 
prevent IOAM Loopback nesting where looped-back packets may be subject to loopback in a 
nested IOAM-Domain. 


4.1.1. Loopback Packet Selection 


If an IOAM encapsulating node incorporates the Loopback flag into all the traffic it forwards, it 
may lead to an excessive amount of looped back packets, which may overload the network and 
the encapsulating node. Therefore, an IOAM encapsulating node that supports the Loopback flag 
MUST support the ability to incorporate the Loopback flag selectively into a subset of the packets 
that are forwarded by it. 


Various methods of packet selection and sampling have been previously defined, such as 
[RFC7014] and [RFC5475]. Similar techniques can be applied by an IOAM encapsulating node to 
apply loopback to a subset of the forwarded traffic. 


The subset of traffic that is forwarded or transmitted with a Loopback flag SHOULD NOT exceed 1/ 
N of the interface capacity on any of the IOAM encapsulating node's interfaces. This requirement 
applies to the total traffic that incorporates a Loopback flag, including traffic that is forwarded by 
the IOAM encapsulating node and probe packets that are generated by the IOAM encapsulating 
node. In this context, N is a parameter that can be configurable by network operators. If there is 
an upper bound, M, on the number of IOAM transit nodes in any path in the network, then 
configuring N such that N >> M (i.e., N is much greater than M) is RECOMMENDED. The rationale 
is that a packet that includes the Loopback flag triggers a looped-back packet from each IOAM 
transit node along the path for a total of M looped-back packets. Thus, if N >> M, then the number 
of looped-back packets is significantly lower than the number of data packets forwarded by the 
IOAM encapsulating node. It is RECOMMENDED that the default value of N satisfies N>100 to be 
used in the absence of explicit operator configuration or if there is no prior knowledge about the 
network topology or size. 


An IOAM-Domain in which the Loopback flag is used MUST be configured such that there is 
expected to be a return path from each of the IOAM transit and IOAM decapsulating nodes; if this 
expectation does not apply, or if the encapsulating node's identity is not available in the 
encapsulation header, then configuration MUST NOT enable the Loopback flag to be set. 
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4.2. Receiving and Processing Loopback 


A Loopback flag that is set indicates to the transit nodes processing this option that they are to 
create a copy of the received packet and send the copy back to the source of the packet. In this 
context, the source is the IOAM encapsulating node and it is assumed that the source address is 
available in the encapsulation header. Thus, the source address of the original packet is used as 
the destination address in the copied packet. If IOAM is used over an encapsulation that does not 
include the address of the encapsulating node, then the transit/decapsulating node does not loop 
back a copy of the original packet. The address of the node performing the copy operation is used 
as the source address; the specific method of source address assignment is encapsulation specific, 
e.g., if an IPv6 encapsulation is used, then the source address can be assigned as specified in 
[RFC6724]. The copy is also truncated, i.e., any payload that resides after the IOAM option(s) is 
removed before transmitting the looped-back packet back towards the encapsulating node. 
Creating the copy that is looped back, and specifically the truncation, may require some 
encapsulation-specific updates in the encapsulation header. The original packet continues 
towards its destination. The L-bit MUST be cleared in the copy of the packet that a node sends 
back towards the source. 


An IOAM node that supports the reception and processing of the Loopback flag MUST support the 
ability to limit the rate of the looped-back packets. The rate of looped-back packets SHOULD be 
limited so that the number of looped-back packets is significantly lower than the number of 
packets that are forwarded by the device. The looped-back data rate SHOULD NOT exceed 1/N of 
the interface capacity on any of the IOAM node's interfaces. Using N>100 is RECOMMENDED. 
Depending on the IOAM node's architecture considerations, the loopback response rate may be 
limited to a lower number in order to avoid overloading the IOAM node. 


4.3. Loopback on the Return Path 


On its way back towards the source, the copied packet is processed like any other packet with 
IOAM information, including adding requested data at each transit node (assuming there is 
sufficient space). 


4.4. Terminating a Looped-Back Packet 


Once the return packet reaches the IOAM-Domain boundary, IOAM decapsulation occurs as with 
any other packet containing IOAM information. Note that the looped-back packet does not have 
the L-bit set. The IOAM encapsulating node that initiated the original loopback packet recognizes 
a received packet as an IOAM looped-back packet by checking the Node ID in the Hop_Lim/ 
node_id field that corresponds to the first hop. If the Node ID and IOAM-Namespace match the 
current IOAM node, it indicates that this is a looped-back packet that was initiated by the current 
IOAM node and processed accordingly. If there is no match in the Node ID, the packet is 
processed like a conventional IOAM-encapsulated packet. 


Note that an IOAM encapsulating node may be either an endpoint (such as an IPv6 host) or a 
switch/router that pushes a tunnel encapsulation onto data packets. In both cases, the 
functionality that was described above avoids IOAM data leaks from the IOAM-Domain. 
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Specifically, if an IOAM looped-back packet reaches an IOAM boundary node that is not the IOAM 
node that initiated the loopback, the node does not process the packet as a loopback; the IOAM 
encapsulation is removed, preventing IOAM information from leaking out from the IOAM- 
Domain. Since the packet does not have any payload, it is terminated. 


5. Active Measurement with IOAM 


Active measurement methods [RFC7799] make use of synthetically generated packets in order to 
facilitate measurement. This section presents use cases of active measurement using the IOAM 
Active flag. 


The Active flag indicates that a packet is used for active measurement. An IOAM decapsulating 
node that receives a packet with the Active flag set in one of its Trace options must terminate the 
packet. The Active flag is intended to simplify the implementation of decapsulating nodes by 
indicating that the packet should not be forwarded further. It is not intended as a replacement 
for existing active OAM protocols, which may run in higher layers and make use of the Active 
flag. 


An example of an IOAM deployment scenario is illustrated in Figure 2. The figure depicts two 
endpoints: a source and a destination. The data traffic from the source to the destination is 
forwarded through a set of network devices, including an IOAM encapsulating node (which 
incorporates one or more IOAM options), a decapsulating node (which removes the IOAM 
options), and optionally one or more transit nodes. The IOAM options are encapsulated in one of 
the IOAM encapsulation types, e.g., [IOAM-NSH] or [IOAM-IPV6-OPTIONS]. 


+-------- + +-------- + +-------- + +-------- + +-------- + 
| | [LOAM | aereee | OAM |e: | IOAM | | | 
+-------- + +-------- + +-------- + +-------- + +-------- + 
| L2/L3 |<===>| L2/L3 |<===>| L2/L3 |<===>| L2/L3 |<===>| L2/L3 | 
+-------- + +-------- + +-------- + +-------- + +-------- + 

Source Encapsulating Transit Decapsulating Destination 

Node Node Node 
Sr IOAM-Domain ----------- > 


Figure 2: Network Using IOAM 


This document focuses on three possible use cases of active measurement using IOAM. These use 
cases are described using the example of Figure 2. 


Mizrahi, et al. Standards Track Page 7 


RFC 9322 IOAM Flags November 2022 


Endpoint active measurement: 
synthetic probe packets are sent between the source and destination, traversing the IOAM- 
Domain. Since the probe packets are sent between the endpoints, these packets are treated as 
data packets by the IOAM-Domain and do not require special treatment at the IOAM layer. 
Specifically, the Active flag is not used in this case and the IOAM layer does not need to be 
aware that an active measurement mechanism is used at a higher layer. 


IOAM active measurement using probe packets within the IOAM-Domain: 
probe packets are generated and transmitted by the IOAM encapsulating node and are 
expected to be terminated by the decapsulating node. IOAM data related to probe packets may 
be exported by one or more nodes along its path by an exporting protocol that is outside the 
scope of this document (e.g., [IOAM-RAWEXPORT]). Probe packets include a Trace Option that 
has its Active flag set, indicating that the decapsulating node must terminate them. The 
specification of these probe packets and the processing of these packets by the encapsulating 
and decapsulating nodes is outside the scope of this document. 


IOAM active measurement using replicated data packets: 
probe packets are created by the encapsulating node by selecting some or all of the en route 
data packets and replicating them. A selected data packet and its (possibly truncated) copy is 
forwarded with one or more IOAM options while the original packet is forwarded normally 
without IOAM options. To the extent possible, the original data packet and its replica are 
forwarded through the same path. The replica includes a Trace Option that has its Active flag 
set, indicating that the decapsulating node should terminate it. The current document defines 
the role of the Active flag in allowing the decapsulating node to terminate the packet, but the 
replication functionality and the functionality of the decapsulating node in this context is 
outside the scope of this document. 


If the volume of traffic that incorporates the Active flag is large, it may overload the network and 
the IOAM node(s) that process the active measurement packet. Thus, the rate of the traffic that 
includes the Active flag SHOULD NOT exceed 1/N of the interface capacity on any of the IOAM 
node's interfaces. Using N>100 is RECOMMENDED. Depending on the IOAM node's architecture 
considerations, the rate of Active-enabled IOAM packets may be limited to a lower number in 
order to avoid overloading the IOAM node. 


6. IANA Considerations 


IANA has allocated the following bits in the "IOAM Trace-Flags" registry as follows: 


Bit1 "Loopback" (L-bit) 
Bit2 "Active" (A-bit) 
This document is specified as the "Reference" in the registry for both bits. 


Note that bit 0 is the most significant bit in the "IOAM Trace-Flags" registry. This bit was allocated 
by [RFC9197] as the 'Overflow' bit. 
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7. Performance Considerations 


Each of the flags that are defined in this document may have performance implications. When 
using the loopback mechanism, a copy of the data packet is sent back to the sender (thus, 
generating more traffic than originally sent by the endpoints). Using active measurement with 
the Active flag requires the use of synthetic (overhead) traffic. 


Each of the mechanisms that use the flags above has a cost in terms of the network bandwidth 
and may potentially load the node that analyzes the data. Therefore, it MUST be possible to use 
each of the mechanisms on a subset of the data traffic; an encapsulating node needs to be able to 
set the Loopback and Active flags selectively in a way that considers the effect on the network 
performance, as further discussed in Sections 4.1.1 and 5. 


Transit and decapsulating nodes that support loopback need to be able to limit the looped-back 
packets (as discussed in Section 4.2) so as to ensure that the mechanisms are used at a rate that 
does not significantly affect the network bandwidth and does not overload the source node in the 
case of loopback. 


8. Security Considerations 


The security considerations of IOAM in general are discussed in [RFC9197]. Specifically, an 
attacker may try to use the functionality that is defined in this document to attack the network. 


IOAM is assumed to be deployed in a restricted administrative domain, thus limiting the scope of 
the threats above and their effect. This is a fundamental assumption with respect to the security 
aspects of IOAM as further discussed in [RFC9197]. However, even given this limited scope, 
security threats should still be considered and mitigated. Specifically, an attacker may attempt to 
overload network devices by injecting synthetic packets that include an IOAM Trace Option with 
one or more of the flags defined in this document. Similarly, an on-path attacker may maliciously 
set one or more of the flags of transit packets. 


Loopback flag: 
an attacker that sets this flag, either in synthetic packets or transit packets, can potentially 
cause an amplification since each device along the path creates a copy of the data packet and 
sends it back to the source. The attacker can potentially leverage the Loopback flag for a DDoS 
attack as multiple devices send looped-back copies of a packet to a single victim. 


Active flag: 
the impact of synthetic packets with the Active flag is no worse than synthetic data packets in 
which the Active flag is not set. By setting the Active flag in en route packets, an attacker can 
prevent these packets from reaching their destination since the packet is terminated by the 
decapsulating device. However, note that an on-path attacker may achieve the same goal by 
changing the destination address of a packet. Another potential threat is amplification; if an 
attacker causes transit switches to replicate more packets than they are intended to replicate 
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(either by setting the Active flag or by sending synthetic packets), then traffic is amplified, 
causing bandwidth degradation. As mentioned in Section 5, the specification of the replication 
mechanism is not within the scope of this document. A specification that defines the 
replication functionality should also address the security aspects of this mechanism. 


Some of the security threats that were discussed in this document may be worse in a wide area 
network in which there are nested IOAM-Domains. For example, if there are two nested IOAM- 
Domains that use loopback, then a looped-back copy in the outer IOAM-Domain may be 
forwarded through another (inner) IOAM-Domain and may be subject to loopback in that (inner) 
IOAM-Domain, causing the amplification to be worse than in the conventional case. 


In order to mitigate the performance-related attacks described in Section 7, it should be possible 
for IOAM-enabled devices to selectively apply the mechanisms that use the flags defined in this 
document to a subset of the traffic and to limit the performance of synthetically generated 
packets to a configurable rate. Specifically, IOAM nodes should be able to: 


e Limit the rate of IOAM packets with the Loopback flag (IOAM encapsulating nodes) as 
discussed in Section 4.1.1. 

e Limit the rate of looped back packets (IOAM transit and decapsulating nodes) as discussed in 
Section 4.2. 

e Limit the rate of IOAM packets with the Active flag (IOAM encapsulating nodes) as discussed 
in Section 5. 


As defined in Section 4, transit nodes that process a packet with the Loopback flag only add a 
single data field and truncate any payload that follows the IOAM option(s), thus significantly 
limiting the possible impact of an amplification attack. 
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